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FINDINGS IN BRIEF 


¢ Students who 
received a personal- 
ized message about 
their potential to 
succeed in AP course- 
work—and who were 
surveyed for this 
study—were 49 per- 
centage points more 
likely to participate in 
AP classes than similar 
students who did not 
receive the message. 


These students also 
became substantially 
more likely to take an 
AP exam and passed 
a higher number of 
exams, making them 
eligible for college 
credit. 


The message had no 
effect on students 
who did not par- 
ticipate in the study's 
survey. 


The results suggest 
that personalized 
messages can bea 
cost-effective way to 
encourage students, 
but might only work 
when combined 
with an effort to call 
students’ attention 
to the content of the 
messages. 
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Small Changes Make a Big Difference: 
How Behavioral Science Improved 
Participation in Advanced Placement 


The Advanced Placement (AP) program has become a key step on the path to attending 
a selective college. In the University of California system, for example, more than 90 
percent of applicants take AP or honors courses in high school (Geiser and Santelices 
2006). However, the College Board estimates that nearly 300,000 high school students 
with the potential to succeed in an AP course graduate every year without ever taking 
one. Furthermore, high-achieving students from disadvantaged backgrounds are more 
likely than other high-achieving students to never take an AP class. 


Behavioral science researchers have shown that making small changes in the way 
information is presented can have a large impact on a person’s decision to participate in a 
program. This issue brief examines how one such change—a personalized message added 
to a test score report—increased student participation in the AP program. 


these criteria to add a personalized message to 
PSAT score reports. Students whose PSAT scores 
met any of the AP Potential criteria received a 


According to the College Board’s research, the 
strongest predictor of success in most AP subjects congratulatory message, informing them that they 
is a student’s performance on the PSAT. Based 

on this finding, the College Board developed “AP 
Potential” criteria to identify students whose PSAT 
scores indicated that they were likely to pass a 


given AP exam.' In 2013, the College Board used 


had “potential for success in at least one AP course” 
(see figure). Students who did not meet any of the 
criteria instead received a general message inviting 
them to speak to their counselor to learn more 
about the AP program. 


Standard and AP Potential Messages 


Students with section 
scores showing potential 
for success in one or more 
AP courses will be directed 


* The PSAT/NMSOT is a great way to get ready for the SAT. 


SAT The best time to take the SAT is spring of junior year. 


Register and practice at sat.org. 
Source: 2013 Sample PSAT Report (College Board 2013). 


to My College QuickStart 
for more information. Other 
students will be advised to 
talk to counselors about how 
to increase preparedness. 


Congratulations, your scores show that you have 
potential for success in at least one AP course! 

Log in to see your full report. AP classes bring college 
to high school to help you get ahead. 


PRINCETON, NJ - ANN ARBOR, MI - CAMBRIDGE, MA - CHICAGO, IL - OAKLAND, CA - SEATTLE, WA 1 
TUCSON, AZ - WASHINGTON, DC - WOODLAWN, MD 


Many high-achieving 
students, particularly 
those from less affluent 
households, miss 

out on educational 
opportunities like AP. 


WHY STUDENTS MIGHT RESPOND 


Many high-achieving students, particularly 
those from less affluent households, miss out on 
educational opportunities like AP. This issue, 
known as academic undermatching, is a serious 
problem in education. Undermatched students 
are less likely to apply to selective colleges, 

even though these institutions can offer them 
academic and financial advantages (Hoxby 

and Avery 2012). They attend less demanding 
colleges or do not attend college at all. 


When students make decisions about their 
education, their choices depend in part on the 
information they have when they assess the 
potential costs and benefits. If they are uncertain 
about any of this information, such as their own 
ability to do well, they might have trouble making 
an optimal decision or might change their minds. 
And evidence suggests that this uncertainty 
matters. For example, research has revealed that 
college students alter their beliefs about their 
academic ability in response to fluctuations 

in their grade point average. These beliefs are 
ultimately associated with decisions like changing 
majors and dropping out of college (Stinebrickner 
and Stinebrickner 2012, 2014; Zafar 2011). 


‘The theory behind an intervention like the AP 
Potential message is that receiving new, high 
quality information should lead students to revise 
their beliefs and potentially make a different 
decision. This theory is supported by a small 

but growing number of studies. For example, 
researchers have found that mailing semi- 
customized information about colleges and their 
net costs led high-achieving students from low- 
income households to apply to and enroll in more 
selective colleges (Hoxby and Turner 2013). 


SURVEY SAYS... 

To test the effects of the AP Potential message 
on participation in AP, this study analyzed school 
records from 10th-grade students in the Oakland 
Unified School District. The study included a 
survey with more than 400 sophomores in one 
large high school in the district, both immediately 
before and after they received their PSAT score 
reports, to examine whether the message affected 
their beliefs. The survey asked students about 
their expected performance on the PSAT, their 
beliefs about their abilities, and their future 
academic plans. Between the pre- and post- 


surveys, the study team distributed the PSAT 
score reports, reviewed the contents of the reports, 
and gave a handout with information to help 
students interpret their results.” 


WHAT THE STUDY REVEALED 

For students with the same PSAT scores, 

those who received the AP Potential message 
generally thought they had performed better than 
originally expected, suggesting the message had 
informational value for them beyond their PSAT 
score alone. These students in turn revised their 
beliefs about their academic ability and future 
academic plans, particularly about the number 

of AP classes they intended to take. 


A regression discontinuity design was used to 
assess whether the AP Potential message affected 
students’ participation in AP courses and exams 
the following school year. Receiving the message 
increased the probability of taking AP courses by 
49 percentage points among surveyed students, 
leading them to enroll in and pass about one more 
AP course, compared with their peers who did 
not receive the message. At the end of the school 
year, these students also were substantially more 
likely to take an AP exam and passed a higher 
number of exams, increasing their eligibility for 
college credit. Being “nudged” into advanced 
coursework by the message had no detrimental 
effects on these students’ academic performance. 
However, for students in other Oakland schools 
who did not participate in the survey and its 
accompanying review of PSAT scores, receiving 
the AP Potential message had no effect. 


The study found no evidence that merely partici- 
pating in the survey affected students’ decisions. 
Instead, participating in the survey, and including 
a review of the PSAT score report, appears to have 
called students’ attention to the AP Potential 
message, leading them to use the information it 
contained. This suggests that receiving the mes- 
sage is not enough—individuals might also need 
to have information brought to their attention for 
it to influence their decision making. 


ADDING IT UP 


Giving qualified students a simple message early 
in their high school careers proved to be a cost- 
effective way to increase their participation in the 
AP program. This type of intervention may be 
especially effective for students from disadvantaged 


backgrounds, who are more likely than their peers —_ with other studies that revealed limited responses 


ra : e to miss out on opportunities like AP. to personalized messages about eligibility for 
People often fail to use 


valuable information 
because their attention 


financial aid and readiness for college (Bettinger 
It is important to note, however, that students et al. 2012; Foote et al. 2015). 
from other area schools who did not participate 


and capacity to process 


+a (teraastiionn aie nal in the survey and accompanying score report We also know that information design matters. The 
review did not respond to the information AP Potential message was relatively inconspicuous 
the message offered. This is a well-established on the PSAT score report (see figure), and students 
behavioral insight—people often fail to use and teachers were unaware of its existence because 
valuable information because their attention and _it was the first year the message had been used. 


capacity to process information are limited. This ‘The survey, including reviewing the PSAT report 
study underscores this point and is consistent and calling attention to the message, may have 
increased the salience of the message. 


The AP Potential Message on the PSAT Score Report 


Name: STUDENT, IMA Year. 2014 = Grade:11 School Code:123456 Optional Code:00 Student Copy 


PSAT/NMSQT’¢ 


Score Report Plus 


Your Scores 


Section Scores Also see 
your projected SAT" scores at 
collegeboard.org/ 
quickstart . 


Score Range Scores in this 
range are similar to yours. 


entite Tiepocerece MES RR AAAAAAAA 


You scored higher than 55% of juniors You scored higher than 57% of juniors. You scored higher than 33% of juniors. 


== National Merit Scholarship Corporation _ Entry Requirements 


{information you provided on yaur answer sheet) 
National Merit Scholarship vouel Me 


Corporation (NMSC) uses 
a Selection Index based 
on PSAT/NMSOT scares 
2s an initial screen of over Year to complete high school and 
1.5 milion students who enter Percentile enroll fulltime in college: 

the National Merit® Scholarship Compares your performance Sali 

Program. (See reverse for more with college-bound juniors. 
information.) 


Eligibility Information 
If your Selection Index places you among the 
'55,000 high scorers who qualify for program 
fecognition, you will be notified next September. 


High school student: US. citizenship: 


‘ears mike spare naradene the SAT and ReadiStep” programs, 


Your Skills Determining the Meaning of Words Number & Operations Grammatical Relationships Between Words 


Theseskilcatecois rp == 
you understand your score and. 9 of 15 questions correct (0 omitted) 12 of 15 questions correct (ompiesoct. soto.ostions correct (1 omitted) 
focus your study efforts before 
you take the SAT Author's Craft: Style, Tone, & Technique Algebra & Functions rases Used to Modify or Compare 
Tol uy skis _——_ : = 
and review suggestions for oF Squestions correct omitted) 3 of S questions correct (0 omitted) of S questions correct(2 omitted) 
inaprovernent based on your Reasoning & Inference Geometry & Measurement Phrases & Clauses 
test performance, visit 
collegeboard.org/ ES eee 
quickstart and sign in using 8 of 10 questions correct (2 omitted) 8 of 10 questions correct (1 omitted) 6 of 10 questions correct (2 omitted) 
your code below. aes he i 
Organization & Ideas Data, Statistics, & Probability Correctly Formed Sentences 
Online Access Code 
| SSE = ] 
A02670146P 2. of 8 questions correct (2 omitted) 6 of 8 questions correct (1 omitted) 110f 6 questions correct (4 omitted) 


Understanding Literary Elements Relationships of Sentences & Paragraphs 


4 of 10 questions correct [0 omitted) 2of 6 questions correct (8 omitted) 


ass és fs fs fs 
Answers ee da/s Fie $ seése/s Fos 
: FES EELEESE £ £ESE/ESESE 
‘See test questions and review Fy Fs XY Fs F FF YT Ss ¢ Fy FF FFs a ae) ae oe, 
explanations of the answers at 7 A ¥ e 25 D ¥ m il iio. 3 ¥ e 2 06C ¥ m 
collegeboard.org/ 2 aa D e 2 2 ¢ — e¢ 2B ¥ h 
quickstart. ae ee) D e 3 3 A+ m 23 & + m 
4ca h A e 4 4 8 Em a ¥ m 
You can also ask your t 
5 5 5 OE! B h 2 D A m 
counselor for a copy of the Fi 
test book you used so you 8 SS 0 6 B De i en 
can review the questions. 7 of Is s 7A Cm a AB e 
8 BA m e 8¢ E h 2 oC A e 
B ¥ m 9 8B m m ae oe 2c OUA Om 
— BOA ~ cv om rm wo aA coh la Dv m 
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tUsorabe fh Had BOA m 8 “aA oh | 2 A Bm “op E m|30 + e 
ees: 3 H Correct Answer Your Answer tote ee 
Scoring cov om c h 7 BY 38 A D e 
© Correct an A ¥ e B h 30 3 D 37 OB et m 
© Omitt — bad m c m 31 4509/2 A 38 04« ¥ m 
‘© Wrong answers to multiple-choice ” a. E ay & 3a_¢ 8 Lt 
‘questions = MINUS 1/4 POINT. 8 Eom c ol m E 
uF ¥ m 45 D B m 
29-38 = NO LOST POINTS. 22 D 8 h 46 A Bom 
ve ¥ m 47 C¢ iE e 
scores an the 20-80 scale. Bak. = Ls 48. E Am 


Information design matters. : 


The AP Potential message (<a 
i i ‘ Next Steps so,yove thinking bout SAT The Psatnwgas saree get endo, eS. 
was relatively inconspicuous Sees Geoeigincrertintne sett Hes ep cg ee SCC Me 
‘on the path to college 
on the score report. i See » Lean mote and see yourfull ests » 1p | Seeger re ecrasters tg mata 
Here are some things collegeboard.org/quickstart apa » Eales eiaeese bring college 
to consider now: Your access code: AO2670146P 


This simple message 
based on standard- 
ized test scores had an 
impact on students’ 
behavior, suggesting 


that existing perfor- 
mance data can be 
used effectively to 
nudge students. 


LOOKING AHEAD 


After 2014, the College Board removed the AP 
Potential message from the PSAT score report. 

It also launched the All In program to provide 
guidance for schools on reaching out to students 
from under-represented groups who meet the AP 
Potential criteria. This study’s results suggest the 
following going forward: 


* The College Board could consider putting the 
message back in the PSAT score report and 
redesigning the report to make the message 
stand out more. 


* — Schools could call students’ attention to the 
message at a potentially minimal cost—for 
example, when teachers distribute the reports 
in class, they could spend 15 minutes going 
over the information (as the survey team did). 


Behavioral science aims to test small 
enhancements designed to change people’s 
actions for the better. This simple message 
based on standardized test scores had an impact 
on students’ behavior, suggesting that existing 
performance data can be used effectively to 
nudge students. Sending a carefully designed 
message can also be much cheaper than other 
types of interventions, such as offering financial 
incentives. In the end, the value of this type of 
intervention is likely to depend on the quality 
and salience of the information it provides. 


For more information, contact 
Dr. Naihobe Gonzalez at NGonzalez@ 
mathematica-mpr.com. 


Read the working paper, “How Learning About 
One’s Ability Affects Educational Investments: 
Evidence from the Advanced Placement 
Program,” on which this brief is based. 


ENDNOTES 

For each AP subject, the College Board deter- 
mines whether a student has at least a 60 percent 
probability of passing that subject’s AP exam, 
based on the PSAT scores most strongly corre- 
lated with this outcome. For example, to meet the 
AP Potential criteria for Calculus BC, a student 
must score at least a 56 on the mathematics sec- 


tion of the PSAT. 


Follow us on: Ei wW in| 


? The handout contained: (1) a conversion table 
to help students predict their future SAT scores 
using their PSAT scores, (2) a table listing the 
SAT score ranges of admitted students at all the 
four-year public colleges in California, and (3) a 
list of the AP courses offered at their school. 
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